Photon Event Distribution Sampling Apparatus and Method

ABSTRACT

Locations of the origins of the photons are acquired from a scanned sample with reference to a scan frame. The location on the sample from which a photon was emitted is inferred from the location of the scan as commanded by a scan drive signal, a feedback signal related to the position of the scan device, or alternatively by the point in time during a scan at which the photon is detected. A position function, e.g., photon probability density, is associated with a photon position. Summing or other processing of photon probability density functions can require fewer photons to converge to an ideal density distribution associated with an image feature than are required using conventional pixel binning. Stored data can be mapped into pixels or voxels of a display or otherwise processed. Original data remains available in the digital storage for post-hoc analysis. Imprecision introduced by the display process need not adversely affect the precision of the collected data.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority from U.S. Provisional PatentApplication Ser. No. 60/573,459, filed May 20, 2004, entitled PHOTONEVENT DISTRIBUTION SAMPLING APPARATUS AND METHOD, the disclosure ofwhich is incorporated herein by reference in its entirety.

FIELD OF INVENTION

This invention concerns high sensitivity imaging apparatus, especiallyoptical systems for examining reflected, fluorescent or chemiluminescentradiation.

BACKGROUND OF THE INVENTION

Detecting photons and producing electronic images from a scanned fieldof view has been performed to produce electronic outputs representingthe field of view of an instrument, such as a laser scanning confocalmicroscope. In this regard, the term “photon” means a unit ofelectromagnetic energy irrespective of its position in the spectrum,e.g. visible or invisible radiation. In quantum physics, a photon ischaracterized as a particle or a wave. The nature of the presentinvention and the manner of its use are not dependent on whether or notthe photon is a particle or a wave.

In one prior art optical detection technique, photons are directed by aconfocal imager in the confocal microscope to be sensed by a detector. Aconfocal imager comprises a point source of light that illuminates aspot within a sample. In order to illuminate an entire sample with aspot, the light source is scanned across a sample by a beam steeringdevice using scanners that are well known in the art. An illuminatedspot is then imaged onto a detector through a pinhole, or “point”aperture. Detectors comprise, for example, avalanche photodiode arraysor photomultiplier tubes.

The light source, the illuminated spot, and the detector have the samefoci; they are placed in conjugate focal-planes. They are therefore“confocal” to each other.

The diameter of the detector aperture is preferably matched to theilluminated spot through the intermediate optics. Because a small spotis illuminated and then detected through a small aperture, only theplane in focus within the specimen is imaged on to the detector. Thedetector produces output pulses indicative of detected photons.

The detector output pulses are processed to provide information such astime-correlated photon-counting histograms and image generation inconventional laser scanning. In conventional imaging systems, however,photons obtained over each of a number of successive, selected equaltime periods defined by a pixel clock are used to generate an intensityvalue assigned to each pixel (two-dimensional area of a portion of animage). Photon counts are binned, that is, accumulated as a group,during each sampling period corresponding to a pixel location of animage display. In this manner, a computer builds up an entire image onepixel at a time to produce an entire two-dimensional image often made upof thousands or multiple millions of pixels. In three-dimensionalimaging, successive two-dimensional layers of a sample are scanned, andthe computer builds up an image comprising voxels.

In producing a conventional image, a scan rate is selected. As scan rateincreases, fewer photons per pixel per scan are accumulated, andintensity of pixels and signal-to-noise ratio therefore decrease. As aresult, prior art pixel-based imaging systems face constraints in scanrate with regard to the quality of output signal to be produced.Physical and mechanical constraints, such as the rate at which a scannercan move, are also present. In addition, the number of photon counts ina sample affects other parameters relating to intensity. Theseparameters include signal-to-noise ratio.

As a result, pixel based scanning typically allows for reducedflexibility in experiment design. Resolution of the location of eachphoton is limited to the dimensions of a pixel or voxel as applicable.The amount of excitation illumination required for output data to reachconvergence of features of sensed images is proportional to the numberof photons that must be produced to provide data sufficient to reachthis convergence. When pixels are of smaller dimension and thereforeprovide fewer photons per scan, samples often must be subjected toexcitation radiation a larger number of times than if the pixels werelarger.

The requirement for greater illumination has functional drawbacks. Inthe subset of applications using fluorescent samples, many moleculesunder test can fluoresce only a limited number of times. At some point,response to excitation radiation ceases, and an effect known asphoto-bleaching occurs. Over illumination also presents anotherdrawback. Where measurements are made in vivo, emission of a photon fromtissue causes free radicals, which can damage cells. Therefore,over-illumination of tissue can result in photo-toxicity.

A limitation of typical prior art techniques is that they are opticallybased. Optically based techniques have an inherent limit of resolutionknown as a diffraction limit, which may be ˜0.6λ, where λ is thewavelength of the illuminating light. The resolving power of a lens isultimately limited by diffraction effects. The lens' aperture is a“hole” that is analogous to a two-dimensional version of the single-slitexperiment. Light passing through it interferes with itself, creating aring-shaped diffraction pattern known as the Airy pattern, that blursthe image. An empirical diffraction limit is given by the Rayleighcriterion: ${\sin\quad\theta} = {1.22\quad\frac{\lambda}{D}}$where θ is the angular resolution, λ is the wavelength of light, and Dis the diameter of the lens. A wave does not have to pass through anaperture to diffract. For example, a beam of light of a finite sizepassing through a lens also undergoes diffraction and spreads indiameter. This effect limits the minimum size d of spot of light formedat the focus of a lens, known as the diffraction limit:${d = {2.44\quad\lambda\quad\frac{f}{a}}},$where λ is the wavelength of the light, f is the focal length of thelens, and a is the diameter of the beam of light, or (if the beam isfilling the lens) the diameter of the lens. Optical techniques do notafford the opportunity to obtain resolution beyond the diffractionlimit.

SUMMARY

The Applicants have discovered that prior art pixel-based imagingtechniques lose information. For example, information typically is lostdue to binning of photons.

The Applicants have also discovered that, in pixel or voxel basedsampling systems, more photons typically are detected than would benecessary if the system did not lose data due to use of the pixelsampling paradigm. In the pixel sampling paradigm, photons collectedduring a predefined pixel clock interval are summed. This summingresults in the loss of spatial and temporal information for individualphotons.

Briefly stated, in accordance with certain embodiments of the presentinvention, an apparatus and a method are provided for use with a scannedsample emitting photons during a scan period in which the location, orsite, of the origin of individual protons, or sets of photons, can bedetermined and recorded. The locations of the sources of individualphotons or sources of photons are acquired in a “pixel-less” manner toyield position information from a detected photon. The locations of theorigins of the photons are acquired with reference to a scan frame thatmay be defined as a single instance of a scan pattern.

In one embodiment, the scanner traverses the scan pattern over one scanperiod. During each successive scan, the scanner may have the samelocation at the same elapsed time from the beginning of the scan period.Therefore, during a scan, a current x-y location of the scanner may havea one-to-one correspondence with a value of a signal associated withscan position. One such signal may be a value of input to a scan driver.Another such signal may be elapsed time from the beginning of a scan. Bymeasuring elapsed time in relation to the beginning of a scan period,position of the scanner may thus be determined. Another such signal maybe values of position feedback from the scan device.

Elapsed time may also be measured from a time the scanner has a knownlocation rather than the beginning of a scan. The time of occurrence ofdetection of each photon is registered. The location on the sample fromwhich a photon was emitted is inferred from the location of the scannerat the time at which the photon is detected.

Certain embodiments measure each photon position by a position functionassociated with the position. One exemplary position function is photonprobability density. The technique then sums or otherwise processes thephoton probability density functions, which can require fewer photons toconverge to an ideal density distribution associated with an imagefeature than are required using conventional pixel binning.Consequently, the same number of photons may be counted to yieldincreased spatial resolution. Sensitivity of measurement can beimproved. Since fewer photons need be detected for a given resolution,less excitation illumination of a sample to produce photons is requiredthan with conventional pixel binning. The technique thus can eliminateor reduce over-illumination of samples and its concomitant adverseeffects.

In certain embodiments, image frames may be constructed by summing thespatial distribution of photons over any user-selected time periodrather than the specific period of a preselected pixel or voxel. Imagescan be displayed in raster space after they are stored digitally.Consequently, any imprecision introduced by the display process need notadversely affect the precision of the collected data. The originalprecise location data remain available in the digital storage.

Since some embodiments can provide high resolution in scan location,these embodiments also can provide high resolution in photon location.In one embodiment, photon location corresponds to an analog signal thatis converted to a digital signal having a preselected number of bits.This number of bits can be selected so that the generated image can bebased in effect on photon location data having a number of bitscorresponding to a resolution of several megapixels per image or more indisplay space. Quality of a displayed image is limited only by thequality of display apparatus and not by the quality of the data.

Other techniques may be utilized to implement differing embodiments ofthe invention. In one form, intervals between detections of individualphotons are recorded. Various points in time displaced by equalintervals may each correspond to a milepost location of a scan. Amilepost location is a predetermined, known location in the scan that isreached at a specific time within the scan period. The location ofdetected photons is preferably calculated by interpolation between themilepost locations. Photons are recorded at a rate that is dependent onthe number of photons detected. Scan rate need not be limited by thenumber of photons expected to be counted in order to achieve aparticular intensity and signal to noise ratio as is necessary in thecase of pixel based sampling.

Alternatively, a signal may be indicative of the x-y position of thescan. The signal can, for example, comprise a monotonically increasingdc signal in which the amplitude of the signal corresponds to a positionof the scanned beam. A detector output indicative of detection of aphoton in one form triggers a sample-and-hold circuit to store theamplitude. The stored amplitude can be recorded. The amplitude can beconverted to a digital value indicative of precise beam position. Thistechnique can therefore generate a signal indicative of time ofdetection of a photon in order to determine photon location. Othertechniques for determining the position of the scan may be used,including use of a clock or a counter activated from the beginning ofthe scan or other milestone.

For further precision, in certain embodiments the effect of variouspossibly interfering phenomena may be reduced or eliminated. Thesephenomena can include sampling delays that may occur in the acquisitionof x-y position information and photon detection, differences betweenthe positions indicated by the signal indicative of scanner positionsand actual scanner positions. Other such phenomena may be the result ofinertia of a scanning element or torque in an arm that rotates to drivea scanning component. Torque can result in different angular positionsof opposite ends of a drive arm. By taking these types of phenomena intoaccount, precision may be even further improved in certain embodimentsof the present invention.

In some embodiments, photon counting may take place at high scan rateswithout the need to account for the number of photon counts in a sample.Image frames in raster space can be generated after the counts areregistered and location data is stored. Thus, using a single data set, adynamic event can be viewed to observe changes occurring over time bycomparing images formed from sequential sample frames. Alternatively,the event can be viewed statically on different time scales.

Availability of complete sets of data in time and space can enablefurther forms of processing of the data, including post-hoc analysis.Post-hoc analysis of the data can allow further analysis of the sampleeven after the sample may become either unavailable or unresponsive tofurther excitation radiation.

In some embodiments, since multiple photons are acquired and statisticalapproaches are used to determine spatial locations of photon clusters,measurement is not limited by the diffraction limit, such as defined bythe Rayleigh criterion (0.6λ/NA) inherent in optical measurements.Consequently, certain embodiments of the present invention can providefor resolution finer than that available from apparatus whose resolutionis limited by the diffraction limit.

The foregoing is a brief summary of various aspects of varyingembodiments of the present invention. This Summary is not exhaustive;additional features and advantages of the invention or variousembodiments will become apparent as this specification proceeds. Inaddition, it is to be understood that embodiments of the invention neednot necessarily address all issues noted in the Background nor includeall features or advantages noted in this Summary or in the balance ofthis specification.

BRIEF DESCRIPTION OF THE DRAWINGS

The preferred and other embodiments are specified in connection with thefollowing drawings in which:

FIG. 1 is a block diagram of a system in which signals indicative ofscan position are produced when a photon is detected;

FIG. 2 is a flow diagram of the preferred method of utilizing the systemshown in FIG. 1;

FIG. 3 consists of FIGS. 3 a and 3 b, in which FIG. 3 a illustratesphotons that have been detected and their positions within one degree offreedom and in which FIG. 3 b illustrates a sum of probability densityfunctions of the photons in FIG. 3 a;

FIG. 4 consists of FIGS. 4 a and 4 b, in which FIG. 4 a illustratesphotons that have been detected associated with an image feature and inwhich FIG. 4 b represents a summation of the probability densityfunctions of the positions of the photons of FIG. 4 a;

FIG. 5 is an illustration of data collected in an embodiment in whichtime intervals between pulses are measured in order to determinelocations of corresponding photons;

FIG. 6, consisting of FIGS. 6 a, 6 b and 6 c, illustrates mapping ofsensed positions into display raster space; and

FIG. 7 is an illustration of beads detected by an embodiment of thepresent invention; the beads are fluorescently labeled and 100 nm inmean diameter. In the present illustration, they were imaged using a40×, 0.95 numerical aperture (NA) microscope objective.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

With reference to FIG. 1, a system 1 constructed in accordance with anembodiment of the present invention is illustrated. A light source 10,generally a laser, provides a light beam 12 to illuminate a sample 14.Light is scanned across the sample 14 by an x-y scanner 16. The x-yscanner 16 provides a scan pattern which may be a periodic linearrepetitive scan, spiral scan, or other scan pattern. Over a scan period,the scanner 16 will scan the light beam 12 over the entire sample 14 ina scan frame. The term scan frame is used to distinguish the frame froman image frame comprising pixels of a sensor which are illuminatedsimultaneously. In alternative embodiments, the scanner 16 is an x-y-zscanner.

The scanner 16 is driven by signals from a drive circuit 18. Many formsof scanners for directing light in a scan pattern are well known in theart. Galvanometer scanners directing rotation of a mirror to movescanned light in first and second degrees of freedom, piezo-actuatedscanners and MEMS (microelectromechanical systems) tip/tilt mirrorscanner are among the scanners that may be used. Since the drive circuit18 provides an input to determine the location of the scan, the drivecircuit 18 produces a signal indicative of a current position of thescanner 16. In one embodiment, the scanner 16 is a non-raster scanner asfurther described in our co-pending U.S. patent application Ser. No.10/795,205, filed Mar. 4, 2004, entitled METHOD AND APPARATUS FORIMAGING USING CONTINUOUS NON-RASTER PATTERNS, published as U.S. PatentApplication No. 20040217270, on Nov. 4, 2004, the disclosure of which isincorporated by reference herein in its entirety.

In a well-known manner, a dichroic mirror 20 directs the light beam 12from the scanner 16 to the sample 14. The dichroic mirror passes lightemitted from the sample 14 to a single photon detector 23. The singlephoton detector 23 may comprise, for example, a photomultiplier tube oran avalanche photodiode or avalanche photodiode array. Avalanchephotodiode arrays reduce well-known adverse effects due to dead timeinherent in avalanche photodiode response. The distribution of eachphoton in space is approximated based on a point spread function of theoptical system embodied in the instrumentation.

In accordance with this embodiment, the drive circuit 18 produces asignal having a value uniquely associated with one position within ascan during each scan interval. This value may be, for example, amonotonically increasing dc value as the scanner 16 progresses throughthe scan pattern. This value is applied to provide a potential level tothe sample and hold circuit 28. When a photon is detected by thesingle-photon detector 23, an output pulse 24 is produced and coupled toapply an input to a discriminator 25. The discriminator 25 produces asquare wave output 26 to provide a clear rising edge and falling edgecoupled to a sample and hold circuit 28, which could comprise, forexample, a well-known RC (resistor-capacitor) circuit. The sample andhold circuit 28 is coupled to sense the signal indicative of actuallocation of the scanned light in the sample being viewed. The sample andhold circuit 28 maintains a potential level which is converted to adigital signal by the analog to digital converter 30. The output of thesample and hold circuit 28 is a signal indicative of the position of thescan.

Outputs of the analog to digital converter 30 may be stored in acomputer 33. As stated above, the current position of a scan alsocorrelates with time elapsed since the beginning of the scan period. Acurrent x-y position, or x-y-z position, of the scanner 16 has aone-to-one correspondence with the elapsed time from the beginning of ascan. Therefore, an alternative signal indicative of the position of thescan is a scan signal indicative of the elapsed time from the beginningof a scan. The time of occurrence of detection of each photon isregistered by the computer 33. Consequently the position of each sensedphoton is determined. As further described below, the computer 33 may beutilized to provide a time associated with each photon detection.Imprecision in the resolution and sample locations obtained during ascan due to an inability of the scanner 16 to faithfully follow thecommand signal can be corrected using an accurate position feedbacksignal from the scanner 16. The computer 33 may also include known videocircuitry to produce an image in response to stored values. The valuesare provided to a video display driver 35 to produce an image on adisplay 37.

In one form, the location, or site, of the origin of every photon withinthe image is determined and the time at which a photon is detected mayalso optionally be recorded. The locations of the sources of individualphotons are acquired with reference to positions of correspondingphotons in a scan frame and without reference to physically definedpixels. A location from which the photon was sensed is the location atwhich the scanner was directed at the time the photon was sensed. Whileit can be desirable to sense every photon to seek to derive the maximumamount of sample information for a given amount of input illumination,images can still be generated in accordance with embodiments of thepresent invention if fewer than all photons are sensed or if groups ofphotons are sensed within a given scan period of a given scan area.

With reference to FIG. 2 the processes of determining photon locationand producing an image comprise a first step 50, at which an operatorplaces the sample 14 in the microscope system 1. In the next step 52,scanning of the light beam 12 over the sample 14 is initiated. Photonlocation is then generated 54 if and when photons are detected by thesingle-photon detector 23. A display of position information 56 may thenbe provided at the display 37. If a scan period is not complete 58,operation continues, and detected photons will again have theirlocations recorded 54. After a scan period, operation proceeds forfurther data processing 58.

Display parameters, e.g. raster locations, are selected 60 to provide aframework so that recorded photon locations in non-raster space can bemapped into raster images on the display 37. Display parameters alsoinclude a grey scale to display density functions. Mapping is requiredsince the recorded data has a finer resolution than the pixels withinthe raster. In other embodiments, in which scans are also taken ofsuccessive depths of the sample 14, a set of sample data is mapped intothree-dimensional voxel space. Probability density functions areaccumulated 62. Probability density functions may be accumulated, forexample, by a distributive or an associative method. In the associativemethod, an intensity value is calculated based on distances to a numbern of nearest photons for each pixel. In a distributive method, eachphoton record is accessed and mapped into one or more raster locations.In this method, both the x and y locations can be sampled to a highdegree of precision. Therefore, the probability density function of asingle photon can make fractional contributions to may pixels. In oneembodiment, they are sampled with 12-bit resolution. This level ofresolution on one axis yields a two-dimensional image resolution of2¹²×2¹², or 16 Megapixels. This level of image resolution may bedescribed as a 16 Megapixel raster space. The accumulation of photondensity functions is continued until processing of an image of a scan iscompleted 64. Another scan may be initiated 66 or operation may becomplete 66.

FIGS. 3 and 4 are illustrations of sensed photons and associatedprobability density functions. FIG. 3 consists of FIGS. 3 a and 3 b. InFIG. 3 a, solid circles each represent a photon that has been detectedand its position within one degree of freedom. Associated with eachphoton is a probability density function. The horizontal extent of FIG.3 defines a region that could, for example, correspond to a pixel.Summing these probability functions yields the solid curve in FIG. 3 b.Summing of photon probability density functions causes the solid curveto converge to an ideal distribution curve represented by the dashedcurve. This summing of probability density functions requires fewerphotons to converge to the ideal density distribution associated with animage feature than are required using conventional pixel binning. As aresult, certain embodiments of the invention can provide greatersensitivity.

FIG. 4 consists of FIGS. 4 a and 4 b. In FIG. 4 a, solid circlesrepresent photons that have been detected that are associated with animage feature. FIG. 4 b represents a summation of the probabilitydensity functions of the positions of the photons of FIG. 4 a. Thecenter of the curve in FIG. 4 b, indicated by an arrow, represents thecentral location of the image feature as determined from the summation.The resolution of the central position obtained in this statisticalmanner is less than diffraction limit of light, 0.6λ/NA. This physicallimit is thus overcome by certain embodiments of the present invention.

FIG. 5 is an illustration of data collected in an embodiment in whichtime intervals between pulses are measured in order to determinelocations of corresponding photons. Data may be collected using theapparatus of FIG. 1. However, rather than having a sample and holdcircuit 28 store a potential level in response to each square wave 26, apulse output indicating time of production of the square wave 26 isproduced. Additionally, the circuit of FIG. 1 produces periodic“milepost” pulses having constant interpulse intervals. Each milepostpulse corresponds to a known position of a scanning beam in the scanpattern of the scanner 16.

In FIG. 5, the horizontal axis represents time. The curve plotted in x-ycoordinates indicates the path of the scanning beam. In the presentexample, the scanning pattern comprises a spiral path. The solid dotsrepresent locations from which a photon is received, and the circlessurrounding each dot represents the point spread function inherent inthe optical system of the confocal microscope system 1. The squaresrepresent milepost beam locations. Each point in time during a scan hasa unique corresponding x-y position. The position of the scan at theoccurrence of each milestone pulse is known. In the present example,photon detection pulses are produced, for example, at the ends ofintervals I₀, I₁, I₂, I₃ and I₄. The computer 33 performs linearinterpolation to determine the location of each detected photon. In thisexample, interpolation is done between times indicated by each one of apair of consecutive milepost beam locations which surround a particularend of an interval. However, interpolation could be done between otherpairs of milepost beam locations surrounding an end of one of theintervals I₀, I₁, I₂, I₃ and I₄. In another alternative form, aplurality of interpolations may be performed respectively between eachof a plurality of pairs of milepost beam locations. The plurality ofinterpolation results could be averaged or otherwise processed. Based onthe point spread function, an estimate of the spatial distribution ismade to generate images as described above.

Certain embodiments can thus provide for measurement of the actuallocations of detected photons. Once the locations are measured, precisevalues indicative of the locations are stored. A complete set oflocation data in time and space is provided, enabling further analysesto be performed. It is not necessary to repeat an experiment in order touse different measurement parameters. Locations can be determined to afar greater degree of precision than is available in currently availabledisplays techniques. The precision is not limited by the physicaldiffraction limit inherent in optical measurements because image framesare constructed by summing photon probability density functions. Thisrequires fewer photons to converge to an ideal density distributionassociated with a feature image than are required using conventionalpixel binning. Resulting increased sensitivity of embodiments of thepresent invention permits reduction or elimination of adverse effects ofover-illumination of samples.

The data obtained as described above may be processed in a number ofdifferent ways. Data thus obtained can readily be rendered in ahistogram format for commonly used analyses, such as fluorescencecorrelation spectroscopy. Alternatively, an interval clock relative tothe timing of a pulsed laser can be triggered by photon detection duringdata acquisition, and fluorescence lifetimes can be analyzed. In anothervariation, detected photons can be categorized according to their energycontent and assigned an appropriate distinguishing color. In addition,it is possible to temporally expand and/or contract a data set acquiredduring a single high-speed acquisition period. Consequently, flexibilityis provided in extracting kinetic information concerning the dynamics ofthe process being imaged.

In some embodiments, repeated scans of a sample allow for comparison ofone scan to another, and corresponding elements of one scan to another.The scans may be consecutive or non-consecutive, and the elements may beimages, portions of images, or photons or sets of photons.Alternatively, positions of photons measured over integrated groups ofscan periods may be compared. With appropriate compensation for noiseand thermal expansion, movements in the subnanometer range may bedetected. FIG. 7 is an illustration of 100 nm beads that have beendetected utilizing the system 1.

Another use of the photon detection data is to produce an image on adisplay. Photon position data is determined to a high degree ofresolution and stored. In order to produce a display, the recordedpositions of photons may be mapped into the raster space of a display.FIG. 6, consisting of FIGS. 6 a, 6 b and 6 c, is useful in understandingembodiments of one method for mapping. In FIG. 6, FIG. 6 a represents anon-raster scan (NRS) pattern, FIG. 6 b is a chart illustrating a“distributive” method of mapping and FIG. 6 c represents an“associative” method of mapping.

With regard to the distributive method, “n” nearest 2-dimensional rasterdisplay pixels are considered as pixels to which one photon scanlocation can be assigned. As in the associative method described below,the influence of each newly acquired sample on nearby raster pixels canbe weighted to the distance between the location of the sample point andthe location of the display pixel (e.g. inversely proportional toproduce linear interpolation). In addition, the time since a pixel waslast updated can be taken into account. Raster (i.e. grid) locations inthe track of the spiral pattern are assigned sampled intensity valuesand weight factors. One suitable weight factor is inversely proportionalto the distance from the actual spiral path. Other functions of distancecan be used. Raster intensity values are then assigned based on theintensities/weights at each location or interpolated from neighboringpixels if no weights/intensities have been stored (a situationillustrated as white grids above). An advantage of this scheme is thatmost of the computation required to produce raster images can beperformed one point at a time, as each sample is acquired. Thedistributive method requires more computational time and is bettersuited for producing a final raster display image, once a completenon-raster sample set has been acquired.

In the associative method, the “n” nearest sampled (i.e. non-raster)points to each 2-dimensional (raster) display pixel are found. Byweighting non-raster sample points inversely proportionally to thedistance between the location of the sample point and the location ofthe raster display pixel, it is possible to rapidly generate an accuraterepresentation of non-raster data with smooth transitions in the rastergrid display. Each raster pixel is assigned a value based on an averageintensity that is weighted. A weighting factor may be inverselyproportional to the distance between the raster location and the spiralsample location. Other functions of distance can be used. In thisexample showing how the intensity values of two pixel locations (pointedto by the collection of arrows) are computed, three nearest spiralsamples are used to make the calculation. In both schemes illustrated InFIGS. 6 a and 6 b, relative weight factors are represented as grayscaleintensity, where darker locations are weighted more heavily.

A significant advantage that can be achieved with some embodiments ofboth mapping methods is that the number of non-raster samples can beindependent of the number of raster display pixels. Thus, if hightemporal resolution (i.e. a high frame rate) is desired, a small numberof samples along a non-raster pattern can produce a roughly uniformdistribution of samples along a raster or non-raster pattern. If highspatial resolution is desired, then more “spirals” as well as datapoints along the spirals can be selected. When tracking rapid dynamicbehaviors or making comparisons with spatial-temporal mathematicalmodels, the non-raster pattern can be used to select just enough spatialresolution while maximizing temporal resolution. The number of samplepoints per image and frame rate can be chosen under software control tobe any values on a continuous scale (i.e. single points can be added orsubtracted). Maximum values are governed only by sinusoidal scanningfrequency and photon detection efficiencies, not by the characteristicsof display devices.

As an example of using a non-raster pattern with an NRS-LSCM, if eachsample required a dwell time of 0.5 microseconds to gather a sufficientnumber of photons and 2000 samples were needed for adequate spatialresolution, then a frame rate of 1000 frames/second could be achieved.Greater frame acquisition rates can be chosen completely under softwarecontrol (i.e. with no modifications to hardware). In these examples, alow number of spiral samples (450 points along the spiral scan) incombination with a low spiral sample to raster pixel ratio haveintentionally been used to simplify illustration of the differentmapping methods.

If desired, the highest possible scan rates can be used withoutconsidering the number of photons being acquired. Once a complete dataset has been acquired, raster space images having desired intensityvalues can be constructed by combining one or more scan frames. Thus,using a single data set, a dynamic event can either be viewed to observechanges occurring over time by comparing images formed from sequentialsample frames, or the event can be viewed statically on different timescales by combining data from more than one sample frame. In the lattercase, the accuracy of parameters obtained for a dynamic event describedat high temporal resolution by one or a few scan frames can be comparedwith those derived from images of the same event obtained over a longerinterval. Useful information can thus be provided as to the interval ofscanning needed to obtain a given level of accuracy.

Storing of the data thus obtained allows for further use of the data.Frame images can be reconstructed and viewed at different temporalrates, thereby permitting compression or expansion of viewing of anoverall data set. The availability of complete data sets in space andtime makes it possible to conduct repeated post-hoc analyses rather thanrepeating an experiment using different measurement parameters. Thissaves cost and reduces inconvenience. Availability of post-hoc analysisguarantees that analysis may be made even when a sample is no longeravailable or is no longer responsive to radiation excitation. Thisfeature provides a number of advantages including the ability to (i)compare conventional images using pixel “bins” with those accumulatedusing probability density functions, and (ii) determine the exactsequence of photonic responses relative to other events.

Another use of photon event sampling is to overcome prior artlimitations of scanning microscopes in achieving required specialresolution and detail in the imaging of a biological sample orinspection of properties of the surface of a material. Doing so withprior art scanning microscope techniques usually involves the frequentneed to utilize high numerical aperture (NA) microscope objectives thatrequire immersion in a fluid medium and close apposition to the sampleor surface to achieve the required spatial resolution and detail. Tominimize sample manipulation and increase throughput, however, it isadvantageous to utilize lower NA objectives having longer workingdistances and with an air interface between the objective and the sampleor surface.

Embodiments of photon event distribution sampling (PEDS) also can beused to counteract the loss of spatial resolution and detail associatedwith use of objectives having lower NA values and longer workingdistances and permits such objectives to be used for the above notedexamples. For example, the 100 nm mean diameter fluorescent beads shownin FIG. 7 were imaged with a 40× objective having a NA value of 0.95using a wavelength of 488 nm. According to the Rayleigh criterion theminimal resolvable object for this objective using this excitationwavelength would have a diameter of 300 nm. Thus, using PEDS, spatialresolutions typically achievable in the past only under specializedconditions using high NA oil-immersion objectives, are achieved withobjectives having both an air interface and a significantly greaterworking distance.

EXAMPLE Application

One application of an embodiment of the present invention is measurementof release of calcium ion, Ca^(2+,) from intracellular sarcoplasmicreticulum (SR) stores in cardiac cells. Ca²⁺ release activates thecontractions of the heart in order to pump blood throughout the body.Ca²⁺ is released through ryanodine receptor (RyR) Ca²⁺ channels presentin SR membranes and information concerning the functional properties ofthese channels, as they exist inside of heart cells, can help inunderstanding how contraction of the heart is activated and regulated.Since RyR channels are present in intracellular membranes, they cannotbe studied using conventional microelectrode-based electrophysiologicaltechniques. However, changes in intracellular Ca²⁺ can be measurednon-invasively by monitoring fluorescence from dyes, such as fluo-3introduced into the cytoplasm of the cell, which increases when Ca²⁺binds to the dye.

Small increases in fluo-3 fluorescence observed in cardiac cells, termedCa²⁺ sparks, are thought to be due to release of Ca²⁺ from a smallnumber of RyR channels. Ca²⁺ sparks have been proposed to representelemental events that are first steps in activating contraction in theheart. As such they have the potential to provide information concerningthe activity and properties of RyR channels in intact cardiac cells. Asituation that complicates relating the properties of Ca²⁺ sparksdirectly to those of RyR channels is that spark properties can beinfluenced by conditions and factors within heart cells not related tothe activity of RyR channels. These factors interact with one anotherand cannot be easily manipulated individually in intact cells and thus,it has proven difficult to assess in a direct manner experimentally howeach factor and condition alters Ca²⁺ spark properties. To date, workersin the field have attempted to use computer modeling and simulations todissect influences by cellular factors and conditions from those relatedto the activity of RyR channels.

An alternative and complementary approach to this problem, is an invitro, optical bilayer system, which permits imaging of fluo-3fluorescence in response to Ca²⁺ flux through single RyR channelsreconstituted into artificial planar lipid bilayer membranessimultaneously with electrical recording of single RyR channel Ca²⁺currents.

Hardware

A suitable confocal microscope included in the system 1 of FIG. 1comprises a BioRad MRC 600 laser scanning confocal microscope system(LSCM) from BioRad Laboratories in Hercules, Calif. In a typicalembodiment, the laser 10 may comprise an ion or solid-state laseremitting in ultraviolet to infrared wavelengths, such as those availablefrom Melles Griot of Rochester, N.Y. and Blue Sky Research of Milpitas,Calif. The scanner 16 may comprise an x-y steering device, includingmirrors mounted on closed loop galvanometers, such as those availablefrom Cambridge Technology, Inc. of Cambridge, Mass. One embodimentutilizes a pair of Cambridge Technology Model 6800 CLG, which are ratedas being capable of scan frequencies of 500-600 Hz over mechanical scanangles of ≦10°. These frequencies were determined for raster scanning(i.e. repeatedly starting and stopping). Scan frequencies of slightlygreater than 1 kHz (limited by safeguard circuitry on the driver card)can be obtained using non-raster scanning and position feedback signalsto establish sample position in non-raster space.

The dichroic mirror 20 is available from Semrock, Inc. of Rochester,N.Y. The laser beam is focused onto a specimen or sample by a device,such as a microscope objective.

Movement of the beam-steering device and scanning of the radiation isdictated by command signals originating from a computer under softwarecontrol and converted to appropriate analog voltages by adigital-to-analog converter with resolution dictated by the spatialresolution requirements of the measurement being made, commonlyavailable from commercial sources, such as National Instruments. Thebeam steering device can be moved in either a raster pattern or innon-raster patterns to scan the electromagnetic radiation across thespecimen or sample. Photons, due to reflected or fluorescent light,originating from a focal plane in the specimen or sample pass throughthe wavelength-selective device and are counted as single events by asingle event detector, such as an avalanche photodiode available fromPerkin Elmer Optoelectronics of Wellesley, Mass. or a photomultipliertube available from Hamamatsu Corporation of Bridgewater, N.J. operatingin a single photon counting mode. A single pulse is sent from thedetector to a discriminator circuit for every photon detected.

The BioRad 600 LSCM can scan a single line in the x dimension in ˜2msec. Therefore a full frame x, y image containing 768×512 pixels or 512x, t-line scans (with each line containing 768 pixels) can be obtainedin ˜1 sec. The rise time of a Ca²⁺ spark in a cardiac cell is ˜8-12msec. Consequently, using this system it is possible to obtain only 6points or less to describe the onset of this event. Single x, t-linescans are employed to achieve the highest scan rates possible. In thisapproach, spatial sampling is collapsed to a single dimension, as thesame line is scanned repeatedly as rapidly as possible. Since Ca²⁺sparks are 4-dimensional events occurring in x, y and z spatialdimensions, as well as in time, this approach also results in suboptimalspatial sampling and potential problems in data interpretation.

One reason for the limited temporal resolution of the MRC 600 in someLSCM systems is that mirrors mounted on separate closed-loopgalvanometers (CLG) are used to scan a laser beam in the x and ydimensions in a raster pattern. This requires that the laser beam beturned around at the beginning and end of each line, which involvesstopping and starting a CLG. Since the shaft of a CLG has significantmass and relatively large mirrors are typically used to accommodatelaser beams whose diameters have been expanded for optical reasons,considerable inertia is involved. The time required to reverse thedirection of the laser beam is a significant portion of the timerequired to scan a single line, which imposes a fundamental limit on thescan rates that can be achieved. In addition, since pixel size incurrent LSCM systems is determined by the pixel clock interval, uniformsampling during a scan requires that the laser beam move at a constantvelocity. Thus, the time required for the CLG to accelerate to aconstant velocity can also impact scanning capabilities.

System Optimization and Desirable Attributes

The discriminator circuit is adjusted to separate pulses caused byphotons from those caused by random noise and each pulse generated by aphoton results in a digital pulse being sent to a sample & hold circuit.Each time a digital pulse is received a corresponding set of voltagesrepresenting the x and y positions of the beam steering device at thetime the photon was detected are retained by the sample & hold circuit.The beam steering device position signals are converted to digitalsignals and passed to the computer by an analog-to-digital converterwith resolution appropriate for the measurement being made, commonlyavailable from commercial sources, such as National Instruments ofAustin, Tex. The position of each photon detected within the imagedomain is transmitted via a display driver to a display device, such asa computer monitor, to form an image of the illuminated region of thespecimen or sample. The display can utilize either a raster ornon-raster scanning format.

The intensity value assigned to each photon in the spatial domain of theimage can be adjusted as a probability density function formulatedrelative to the point-spread-function of the illuminating radiation andthe probability of exciting emission from a fluorophore positionedwithin the illumination point spread function. The focal plane in thespecimen or sample being illuminated by the electromagnetic radiation isselected under computer control via focus control circuitry controllingthe position in the z-axis at which the radiation is focused in thespecimen or sample. This control is currently typically implemented viaa serial (USB) port interface with the focus control circuitry. As isthe case with x and y position signals, voltage-indicated z positionsmay also be passed to the sample & hold circuit 28 and subsequently tothe computer 33 via an analog-to-digital converter channel. It should benoted that algorithms commonly used to eliminate photons originatingfrom above and below the focal plane and enhance images obtained withthe approach presently described.

The NRS-LSCM thus described offers particular advantages for imagingdynamic processes. Such dynamic processes include change inintracellular Ca²⁺ (Ca²⁺ sparks and waves) involved in tissue activationand intracellular signaling, changes in membrane potential in excitabletissues (e.g. heart and brain), or the spread of activation within theGI tract. Images of these events, as well as of many other cellularprocesses, share in common the fact that they contain relatively lowspatial frequencies. Therefore, relatively low sampling frequencies (andconsequently high scan rates) can be used to establish their propertiesin an adequate manner. Embodiments of the present invention are designedto be capable of imaging events involving intermediate-to-low spatialfrequencies at maximal possible (photon-limited) sampling rates.Moreover, because spatial resolution capabilities have not beensacrificed to obtain greater temporal resolution, the NRS-LSCM systemwill equal the data collection rate performance of systems for imagingsamples containing high spatial frequencies, where greater samplingrates (and lower scan speeds) are used.

In the embodiments described above, the scan format is a type ofnon-raster scan identified as a spiral scan. Other scan formats may beutilized, including other raster and non-raster scan formats.

It is to be understood that the foregoing is a description of preferredand other embodiments. The foregoing description therefore is not to beconstrued as itself limiting of the scope of the invention.

1. A data generating method comprising: A) scanning a sample during ascan period over a scan pattern comprising a plurality of positions; B)detecting a packet of electromagnetic energy received from the sample inresponse to the scanning during the scan; and C) associating the packetof electromagnetic energy with a first position in the scan pattern. 2.The data generating method of claim 1 wherein the detecting step (B)further comprises detecting a plurality of subsets of packets ofelectromagnetic energy received from the sample in response to thescanning during said scan period of time; and wherein the associatingstep (C) further comprises associating each subset of packets among theplurality of subsets of packets of electromagnetic energy with acorresponding position during the scan period.
 3. The data generatingmethod of claim 2 wherein each subset of packets consists of one packetof electromagnetic energy and the detecting step (B) further comprisesproducing an output indicative of detection of the photon.
 4. A datagenerating method according to claim 3, further comprising detectingphotons over successive scan periods and producing data in response todetection.
 5. A data generating method according to claim 3, furthercomprising combining data produced over successive scan periods inaccordance with a function.
 6. A data generating method according toclaim 5, comprising mapping the data into at least one image.
 7. A datagenerating method according to claim 6, comprising associating each scanposition with a subdivision of the image.
 8. A data generating methodaccording to claim 7, further comprising establishing subdivisions ofthe image each corresponding to a pixel of a display.
 9. A datagenerating method according to claim 8 further comprising producing apixel signal in correspondence with data assigned to each of a pluralityof respective pixels and providing said pixel signals to a displaydriver.
 10. A data generating method according to claim 3, furthercomprising sensing the individual photons with a photon detector.
 11. Adata generating method according to claim 7, wherein the scan isthree-dimensional.
 12. A data generating method according to claim 11,further comprising subdividing each image into voxels.
 13. A datagenerating method according to claim 6, further comprising providing afluorescent sample and scanning said sample with excitation radiation.14. A data generating method according to claim 6, further comprisingproviding a reflecting sample and scanning said sample with excitationradiation.
 15. A data generating method according to claim 13, whereindetecting comprises viewing photons from the sample with a confocalmicroscope and focusing said photons on a photon detector.
 16. A datagenerating method according to claim 13, wherein detecting comprisesviewing photons from the sample with a confocal microscope and focusingsaid photons on a photon detector.
 17. A data generating methodaccording to claim 5, further comprising generating a position functionfor detected photons.
 18. A data generating method according to claim 17wherein generating a position function comprises generating aprobability density.
 19. A data generating method according to claim 18,further comprising summing probability densities, determining spatialdistribution, and resolving image features.
 20. A data generating methodaccording to claim 19, further comprising summing spatial distributionover a predetermined time period.
 21. An imaging system comprising: a) asensor to detect a photon emitted by a sample subjected to a periodicscan having a scan pattern and a scan period, the scan pattern includinga plurality of positions; b) a scan circuit providing a scan signalhaving a value indicative of scan position; and c) a register circuit toassociate detection of a photon with a corresponding value of the scansignal.
 22. An imaging system according to claim 21, wherein saidregister circuit responds detection of each of a plurality of photonswithin a scan period.
 23. An imaging system according to claim 22,wherein said position circuit provides position outputs each indicativeof a photon position.
 24. An imaging system according to claim 23,wherein said register circuit further comprises storage to storeposition outputs over a plurality of scan periods.
 25. An imaging systemaccording to claim 24, further comprising a video display driverresponsive to outputs accessed from said storage to map said outputsinto a display raster space.
 26. An imaging system according to claim24, comprising a three-dimensional scanner and comprising a videodisplay driver to map outputs accessed from said storage means to mapsaid outputs into voxels.
 27. An imaging system according to claim 24,wherein said scan signal comprises a signal indicative of a scan driversignal determining scan position.
 28. An imaging system according toclaim 24, wherein said scan signal comprises a signal indicative ofelapsed time within a scan period.
 29. An imaging system according toclaim 24, further comprising a confocal microscope and a photondetector, said photon detector having radiation focused thereon by saidconfocal microscope and producing at least one image based on aplurality of photons.
 30. A computer-readable storage medium comprisinginstructions which when executed on a computer cause the computer toperform the steps of: A) scanning a sample during a scan period over ascan pattern comprising a plurality of positions; B) detecting a packetof electromagnetic energy received from the sample in response to thescanning during the scan; and C) associating the packet ofelectromagnetic energy with a first position in the scan pattern.
 31. Acomputer-readable storage medium according to claim 30, wherein thedetecting step (B) further comprises detecting a plurality of subsets ofpackets of electromagnetic energy received from the sample in responseto the scanning during said scan period of time; and wherein theassociating step (C) further comprises associating each subset ofpackets among the plurality of subsets of packets of electromagneticenergy with a corresponding position during the scan period.
 32. Acomputer-readable storage medium according to claim 31, wherein eachsubset of packets consists of one packet of electromagnetic energy andthe detecting step (B) further comprises producing an output indicativeof detection of the photon.
 33. A computer-readable storage mediumaccording to claim 31, further comprising instructions for combiningdata produced over successive scan periods in accordance with afunction.
 34. A computer-readable storage medium according to claim 32,comprising instructions for mapping the data into an image.
 35. Acomputer-readable storage medium according to claim 33, comprisinginstructions for associating each scan position with a subdivision ofthe image.
 36. A computer-readable storage medium according to claim 34,further comprising instructions for establishing subdivisions of theimage each corresponding to a pixel of a display.
 37. Acomputer-readable storage medium according to claim 35 furthercomprising instructions for producing a pixel signal in correspondencewith data assigned to each of a plurality of respective pixels andproviding said pixel signals to a display driver.
 38. Acomputer-readable storage medium according to claim 35, furthercomprising instructions for responding to sensing of the individualphotons with a photon detector.
 39. A computer-readable storage mediumaccording to claim 36, wherein the scan is three-dimensional and furthercomprising instructions for subdividing each image into voxels.
 40. Acomputer-readable storage medium according to claim 38, furthercomprising instructions for generating a position function for detectedphotons.
 41. A computer-readable storage medium according to claim 38wherein generating a position function comprises generating aprobability density.
 42. A computer-readable storage medium according toclaim 39, further comprising instructions for summing probabilitydensities, determining spatial distribution, and resolving imagefeatures.
 43. A computer-readable storage medium according to claim 40,further comprising instructions for producing an image frame by summingspatial distribution over a user-selected time period.